Skip to content

DRAFT: Rebase av2enc on main - #5331

Open
yeqing-wu wants to merge 205 commits into
AOMediaCodec:av2-encfrom
yeqing-wu:rebase_av2enc_on_main
Open

DRAFT: Rebase av2enc on main#5331
yeqing-wu wants to merge 205 commits into
AOMediaCodec:av2-encfrom
yeqing-wu:rebase_av2enc_on_main

Conversation

@yeqing-wu

Copy link
Copy Markdown
Contributor

Please don't squash — we need to preserve the commit history, so rebase only.

y-guyon and others added 30 commits July 6, 2026 06:05
See https://github.com/zizmorcore/zizmor.

We pin GitHub Actions used in CI workflows to git hashes instead of
version tags.

Similar change was also made to libavif's CI workflows in PR
AOMediaCodec/libavif#1515 at the suggestion of Google's security team.
The rationale is described in issue AOMediaCodec/libavif#1514.

<details>

<summary><code>zizmor --fix=all --gh-token=$(gh auth token)
.github/workflows</code> output</summary>

```
 INFO zizmor: \U0001f308 zizmor v1.25.2
 WARN collect_inputs: zizmor::registry::input: failed to validate file://.github/workflows/pull_request.yaml as workflow: input does not match expected validation schema
 WARN collect_inputs: zizmor::registry::input: failed to validate file://.github/workflows/nightly.yaml as workflow: input does not match expected validation schema
 INFO audit: zizmor: \U0001f308 completed .github/workflows/build-job-reusable.yaml
 INFO audit: zizmor: \U0001f308 completed .github/workflows/common-builds-reusable.yaml
 INFO audit: zizmor: \U0001f308 completed .github/workflows/common-test-data-reusable.yaml
 INFO audit: zizmor: \U0001f308 completed .github/workflows/sanitizer-job-reusable.yaml
warning[artipacked]: credential persistence through GitHub Actions artifacts
  --> .github/workflows/build-job-reusable.yaml:61:9
   |
61 |         - uses: actions/checkout@v5
   |  _________^
62 | |         with:
63 | |           fetch-depth: 50
64 | |           lfs: true
   | |___________________^ does not set persist-credentials: false
   |
   = note: audit confidence \u2192 Low
   = note: this finding has an auto-fix

error[template-injection]: code injection via template expansion
  --> .github/workflows/build-job-reusable.yaml:72:18
   |
72 |         run: ${{ inputs.before-script }}
   |         ---      ^^^^^^^^^^^^^^^^^^^^ may expand into attacker-controllable code
   |         |
   |         this run block
   |
   = note: audit confidence \u2192 High
   = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
  --> .github/workflows/build-job-reusable.yaml:61:15
   |
61 |       - uses: actions/checkout@v5
   |               ^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
   |
   = note: audit confidence \u2192 High
   = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
  --> .github/workflows/build-job-reusable.yaml:74:15
   |
74 |       - uses: actions/cache/restore@v5
   |               ^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
   |
   = note: audit confidence \u2192 High
   = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
   --> .github/workflows/build-job-reusable.yaml:182:15
    |
182 |       - uses: actions/cache/save@v5
    |               ^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
    |
    = note: audit confidence \u2192 High
    = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
   --> .github/workflows/build-job-reusable.yaml:188:15
    |
188 |         uses: actions/upload-artifact@v7
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
    |
    = note: audit confidence \u2192 High
    = note: this finding has an auto-fix

error[unpinned-images]: unpinned image references
  --> .github/workflows/build-job-reusable.yaml:51:18
   |
51 |       image: ${{ inputs.container-image }}
   |                  ^^^^^^^^^^^^^^^^^^^^^^ container image may be unpinned
   |
   = note: audit confidence \u2192 Low

warning[artipacked]: credential persistence through GitHub Actions artifacts
   --> .github/workflows/common-builds-reusable.yaml:305:9
    |
305 |         - uses: actions/checkout@v5
    |  _________^
306 | |         with:
307 | |           fetch-depth: 50
308 | |           lfs: true
    | |___________________^ does not set persist-credentials: false
    |
    = note: audit confidence \u2192 Low
    = note: this finding has an auto-fix

warning[artipacked]: credential persistence through GitHub Actions artifacts
   --> .github/workflows/common-builds-reusable.yaml:356:9
    |
356 |         - uses: actions/checkout@v5
    |  _________^
357 | |         with:
358 | |           fetch-depth: 50
359 | |           lfs: true
    | |___________________^ does not set persist-credentials: false
    |
    = note: audit confidence \u2192 Low
    = note: this finding has an auto-fix

warning[artipacked]: credential persistence through GitHub Actions artifacts
   --> .github/workflows/common-builds-reusable.yaml:468:9
    |
468 |         - uses: actions/checkout@v5
    |  _________^
469 | |         with:
470 | |           fetch-depth: 50
471 | |           fetch-tags: true
472 | |           lfs: true
    | |___________________^ does not set persist-credentials: false
    |
    = note: audit confidence \u2192 Low
    = note: this finding has an auto-fix

warning[excessive-permissions]: overly broad permissions
  --> .github/workflows/common-builds-reusable.yaml:23:3
   |
23 | /   build-generic-gnu:
24 | |     name: Build (generic-gnu)
25 | |     uses: ./.github/workflows/build-job-reusable.yaml
26 | |     if: (!cancelled())
...  |
45 | |           - inspection-accounting
46 | |           - no-examples
   | |                       ^
   | |                       |
   | |_______________________this job
   |                         default permissions used due to no permissions: block
   |
   = note: audit confidence \u2192 Medium

warning[excessive-permissions]: overly broad permissions
  --> .github/workflows/common-builds-reusable.yaml:48:3
   |
48 | /   build-x86_64-linux-gcc:
49 | |     name: Build (x86_64-linux-gcc)
50 | |     uses: ./.github/workflows/build-job-reusable.yaml
51 | |     if: (!cancelled())
...  |
72 | |           - debug
73 | |           - enable-12bit-profile
   | |                                ^
   | |                                |
   | |________________________________this job
   |                                  default permissions used due to no permissions: block
   |
   = note: audit confidence \u2192 Medium

warning[excessive-permissions]: overly broad permissions
  --> .github/workflows/common-builds-reusable.yaml:75:3
   |
75 | /   build-entropy-stats:
76 | |     name: Build (entropy-stats)
77 | |     uses: ./.github/workflows/build-job-reusable.yaml
78 | |     if: (!cancelled())
...  |
90 | |         avm-build-config:
91 | |           - entropy-stats
   | |                         ^
   | |                         |
   | |_________________________this job
   |                           default permissions used due to no permissions: block
   |
   = note: audit confidence \u2192 Medium

warning[excessive-permissions]: overly broad permissions
   --> .github/workflows/common-builds-reusable.yaml:93:3
    |
 93 | /   build-bitstream-mismatch-debug:
 94 | |     name: Build (bitstream-mismatch-debug)
 95 | |     uses: ./.github/workflows/build-job-reusable.yaml
 96 | |     if: (!cancelled())
...   |
105 | |         avm-build-config:
106 | |           - bitstream-mismatch-debug
    | |                                    ^
    | |                                    |
    | |____________________________________this job
    |                                      default permissions used due to no permissions: block
    |
    = note: audit confidence \u2192 Medium

warning[excessive-permissions]: overly broad permissions
   --> .github/workflows/common-builds-reusable.yaml:108:3
    |
108 | /   build-enable-12bit-profile:
109 | |     name: Build (enable-12bit-profile)
110 | |     uses: ./.github/workflows/build-job-reusable.yaml
111 | |     if: (!cancelled())
...   |
120 | |         avm-build-config:
121 | |           - enable-12bit-profile
    | |                                ^
    | |                                |
    | |________________________________this job
    |                                  default permissions used due to no permissions: block
    |
    = note: audit confidence \u2192 Medium

warning[excessive-permissions]: overly broad permissions
   --> .github/workflows/common-builds-reusable.yaml:123:3
    |
123 | /   build-x86-linux-gcc:
124 | |     name: Build (x86-linux-gcc)
125 | |     uses: ./.github/workflows/build-job-reusable.yaml
126 | |     if: (!cancelled())
...   |
147 | |           - no-examples
148 | |           - debug
    | |                 ^
    | |                 |
    | |_________________this job
    |                   default permissions used due to no permissions: block
    |
    = note: audit confidence \u2192 Medium

warning[excessive-permissions]: overly broad permissions
   --> .github/workflows/common-builds-reusable.yaml:150:3
    |
150 | /   build-aarch64-linux-gcc:
151 | |     name: Build (aarch64-linux-gcc)
152 | |     uses: ./.github/workflows/build-job-reusable.yaml
153 | |     if: (!cancelled())
...   |
173 | |           - inspection-accounting
174 | |           - no-examples
    | |                       ^
    | |                       |
    | |_______________________this job
    |                         default permissions used due to no permissions: block
    |
    = note: audit confidence \u2192 Medium

warning[excessive-permissions]: overly broad permissions
   --> .github/workflows/common-builds-reusable.yaml:176:3
    |
176 | /   build-armv7-linux-gcc:
177 | |     name: Build (armv7-linux-gcc)
178 | |     uses: ./.github/workflows/build-job-reusable.yaml
179 | |     if: (!cancelled())
...   |
199 | |           - inspection-accounting
200 | |           - no-examples
    | |                       ^
    | |                       |
    | |_______________________this job
    |                         default permissions used due to no permissions: block
    |
    = note: audit confidence \u2192 Medium

warning[excessive-permissions]: overly broad permissions
   --> .github/workflows/common-builds-reusable.yaml:202:3
    |
202 | /   build-ppc-linux-gcc:
203 | |     name: Build (ppc-linux-gcc)
204 | |     uses: ./.github/workflows/build-job-reusable.yaml
205 | |     if: (!cancelled())
...   |
226 | |           - no-examples
227 | |           - debug
    | |                 ^
    | |                 |
    | |_________________this job
    |                   default permissions used due to no permissions: block
    |
    = note: audit confidence \u2192 Medium

warning[excessive-permissions]: overly broad permissions
   --> .github/workflows/common-builds-reusable.yaml:229:3
    |
229 | /   build-x86-mingw-gcc:
230 | |     name: Build (x86-mingw-gcc)
231 | |     uses: ./.github/workflows/build-job-reusable.yaml
232 | |     if: (!cancelled())
...   |
251 | |           - nasm
252 | |           - no-examples
    | |                       ^
    | |                       |
    | |_______________________this job
    |                         default permissions used due to no permissions: block
    |
    = note: audit confidence \u2192 Medium

warning[excessive-permissions]: overly broad permissions
   --> .github/workflows/common-builds-reusable.yaml:254:3
    |
254 | /   build-x86_64-mingw-gcc:
255 | |     name: Build (x86_64-mingw-gcc)
256 | |     uses: ./.github/workflows/build-job-reusable.yaml
257 | |     if: (!cancelled())
...   |
276 | |           - nasm
277 | |           - no-examples
    | |                       ^
    | |                       |
    | |_______________________this job
    |                         default permissions used due to no permissions: block
    |
    = note: audit confidence \u2192 Medium

warning[excessive-permissions]: overly broad permissions
   --> .github/workflows/common-builds-reusable.yaml:279:3
    |
279 | /   build-x86_64-clang:
280 | |     name: Build (x86_64-clang)
281 | |     uses: ./.github/workflows/build-job-reusable.yaml
282 | |     if: (!cancelled())
...   |
294 | |           - debug
295 | |           - release
    | |                   ^
    | |                   |
    | |___________________this job
    |                     default permissions used due to no permissions: block
    |
    = note: audit confidence \u2192 Medium

error[unpinned-uses]: unpinned action reference
   --> .github/workflows/common-builds-reusable.yaml:305:15
    |
305 |       - uses: actions/checkout@v5
    |               ^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
    |
    = note: audit confidence \u2192 High
    = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
   --> .github/workflows/common-builds-reusable.yaml:329:15
    |
329 |         uses: actions/upload-artifact@v7
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
    |
    = note: audit confidence \u2192 High
    = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
   --> .github/workflows/common-builds-reusable.yaml:356:15
    |
356 |       - uses: actions/checkout@v5
    |               ^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
    |
    = note: audit confidence \u2192 High
    = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
   --> .github/workflows/common-builds-reusable.yaml:363:15
    |
363 |       - uses: actions/cache/restore@v5
    |               ^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
    |
    = note: audit confidence \u2192 High
    = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
   --> .github/workflows/common-builds-reusable.yaml:430:15
    |
430 |       - uses: actions/cache/save@v5
    |               ^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
    |
    = note: audit confidence \u2192 High
    = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
   --> .github/workflows/common-builds-reusable.yaml:436:15
    |
436 |         uses: actions/upload-artifact@v7
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
    |
    = note: audit confidence \u2192 High
    = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
   --> .github/workflows/common-builds-reusable.yaml:468:15
    |
468 |       - uses: actions/checkout@v5
    |               ^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
    |
    = note: audit confidence \u2192 High
    = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
   --> .github/workflows/common-builds-reusable.yaml:476:15
    |
476 |       - uses: actions/cache/restore@v5
    |               ^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
    |
    = note: audit confidence \u2192 High
    = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
   --> .github/workflows/common-builds-reusable.yaml:505:15
    |
505 |       - uses: actions/cache/save@v5
    |               ^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
    |
    = note: audit confidence \u2192 High
    = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
   --> .github/workflows/common-builds-reusable.yaml:511:15
    |
511 |         uses: actions/upload-artifact@v7
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
    |
    = note: audit confidence \u2192 High
    = note: this finding has an auto-fix

error[unpinned-images]: unpinned image references
   --> .github/workflows/common-builds-reusable.yaml:303:18
    |
303 |       image: ${{ inputs.container-image }}
    |                  ^^^^^^^^^^^^^^^^^^^^^^ container image may be unpinned
    |
    = note: audit confidence \u2192 Low

error[unpinned-images]: unpinned image references
   --> .github/workflows/common-builds-reusable.yaml:342:18
    |
342 |       image: ${{ inputs.container-image }}
    |                  ^^^^^^^^^^^^^^^^^^^^^^ container image may be unpinned
    |
    = note: audit confidence \u2192 Low

error[unpinned-images]: unpinned image references
   --> .github/workflows/common-builds-reusable.yaml:462:18
    |
462 |       image: ${{ inputs.container-image }}
    |                  ^^^^^^^^^^^^^^^^^^^^^^ container image may be unpinned
    |
    = note: audit confidence \u2192 Low

warning[artipacked]: credential persistence through GitHub Actions artifacts
  --> .github/workflows/common-test-data-reusable.yaml:29:9
   |
29 |         - uses: actions/checkout@v5
   |  _________^
30 | |         with:
31 | |           fetch-depth: 50
32 | |           lfs: true
   | |___________________^ does not set persist-credentials: false
   |
   = note: audit confidence \u2192 Low
   = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
  --> .github/workflows/common-test-data-reusable.yaml:29:15
   |
29 |       - uses: actions/checkout@v5
   |               ^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
   |
   = note: audit confidence \u2192 High
   = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
  --> .github/workflows/common-test-data-reusable.yaml:42:15
   |
42 |         uses: actions/upload-artifact@v7
   |               ^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
   |
   = note: audit confidence \u2192 High
   = note: this finding has an auto-fix

error[unpinned-images]: unpinned image references
  --> .github/workflows/common-test-data-reusable.yaml:25:18
   |
25 |       image: ${{ inputs.container-image }}
   |                  ^^^^^^^^^^^^^^^^^^^^^^ container image may be unpinned
   |
   = note: audit confidence \u2192 Low

warning[artipacked]: credential persistence through GitHub Actions artifacts
  --> .github/workflows/sanitizer-job-reusable.yaml:64:9
   |
64 |         - uses: actions/checkout@v5
   |  _________^
65 | |         with:
66 | |           fetch-depth: 50
67 | |           lfs: true
   | |___________________^ does not set persist-credentials: false
   |
   = note: audit confidence \u2192 Low
   = note: this finding has an auto-fix

error[template-injection]: code injection via template expansion
  --> .github/workflows/sanitizer-job-reusable.yaml:98:18
   |
98 |         run: ${{ inputs.before-script }}
   |         ---      ^^^^^^^^^^^^^^^^^^^^ may expand into attacker-controllable code
   |         |
   |         this run block
   |
   = note: audit confidence \u2192 High
   = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
  --> .github/workflows/sanitizer-job-reusable.yaml:64:15
   |
64 |       - uses: actions/checkout@v5
   |               ^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
   |
   = note: audit confidence \u2192 High
   = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
  --> .github/workflows/sanitizer-job-reusable.yaml:75:15
   |
75 |         uses: actions/download-artifact@v8
   |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
   |
   = note: audit confidence \u2192 High
   = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
  --> .github/workflows/sanitizer-job-reusable.yaml:80:15
   |
80 |         uses: actions/download-artifact@v8
   |               ^^^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
   |
   = note: audit confidence \u2192 High
   = note: this finding has an auto-fix

error[unpinned-uses]: unpinned action reference
   --> .github/workflows/sanitizer-job-reusable.yaml:230:15
    |
230 |         uses: actions/upload-artifact@v7
    |               ^^^^^^^^^^^^^^^^^^^^^^^^^^ action is not pinned to a hash (required by blanket policy)
    |
    = note: audit confidence \u2192 High
    = note: this finding has an auto-fix

error[unpinned-images]: unpinned image references
  --> .github/workflows/sanitizer-job-reusable.yaml:51:18
   |
51 |       image: ${{ inputs.container-image }}
   |                  ^^^^^^^^^^^^^^^^^^^^^^ container image may be unpinned
   |
   = note: audit confidence \u2192 Low

60 findings (14 suppressed, 28 unsafe fixes): 0 informational, 0 low, 18 medium, 28 high

Fix Summary
Successfully applied fixes to 4 files:
  .github/workflows/common-builds-reusable.yaml: 13 fixes
  .github/workflows/sanitizer-job-reusable.yaml: 6 fixes
  .github/workflows/build-job-reusable.yaml: 6 fixes
  .github/workflows/common-test-data-reusable.yaml: 3 fixes
```

</details>
…OMediaCodec#5083)

Initialize `ref_costs_single` for INTRA and TIP frame indices to
`INT_MAX` when they are not allowed.

Previously, these elements were left uninitialized in
`estimate_ref_frame_costs` if the respective modes were not allowed, but
they were still being read and passed to other functions in
`av2_rd_pick_inter_mode_sb`, triggering MSAN warnings.

Fixes AOMediaCodec#5081
The default value for this feature is 1, fix the comment.
prune_split_with_ml/prune_none_with_ml should be reset to 0. Set
prune_split_ml_level/prune_split_ml_level_inter to - 1.
When the encoder is run in error-resilient mode
(`--global-error-resilient=1`), switch frames are emitted without their
reference frames being marked as restricted. This produces a bitstream
that is not AV2-compliant and that decoders reject.
  
This MR sets `restricted_prediction_switch` for error-resilient switch
frames as well, so switch frames generated in that mode carry the
correct restricted-reference semantics.

In error-resilient mode (`--global-error-resilient=1`), the current AVM
encoder converts alt-ref–type frames (which are reordered/future frames)
into switch frames. When such a switch is left unconstrained
(non-restricted), the encoder can assign it a low display order hint
(e.g. 1) while a frame with a higher display order hint (e.g. 4) has
already been output, producing a non-monotonic output order. Because a
non-restricted switch does not reset the OrderHint counter, this
violates the conformance constraint below (AV2 spec §7.3.6, "Coded
extended layer unit"):

``
If monotonic_output_order_flag is equal to 0, it is a requirement of
bitstream conformance that within a coded video sequence, for a given
value of obu_xlayer_id and obu_mlayer_id, if a coded output frame unit X
has an associated OrderHint value equal to ohX, there shall not be a
coded output frame unit Y in the same extended layer and embedded layer
that appears later than X in output order and has an associated
OrderHint value less than or equal to ohX, unless a switch frame with
restricted_prediction_switch equal to 1 appears between X and Y in
coding order.
``
``
NOTE: The value of OrderHint is reset at the start of a new coded video
sequence and at a switch frame with restricted_prediction_switch equal
to 1. In both cases, the OrderHint counter is effectively restarted,
allowing OrderHint values to be reused in subsequent coded output frame
units.
``
…iaCodec#5097)

The `ref_frame_sign_bias` array in the `AV2Common` structure is
populated in `av2_setup_frame_sign_bias()` but never used. This patch
cleans up the unused array and the associated code.
For tests with dynamic layer dropping:
CDF_AVERAGING should be set to 0/off, to avoid DPB desync.

Also for the MultiLayerTest2Embedded2TemporaSframe test
enable_explicit_ref_frame_map_ should be set to true.

Failures were seen at the lower speed settings (e.g., speed 2), not at
the speed 5 setting used in the test.
In the reference commit, `cm->ref_frame_side[]` and
`cm->ref_frame_relative_dist[]` were computed in two separate paths:
either inside `av2_setup_motion_field()` when `allow_ref_frame_mvs` is
true, or in `av2_setup_ref_frame_sides()`. In this patch,
`av2_setup_ref_frame_sides()` is called unconditionally to populate
these reference frame arrays, thus removing the duplicate computations
inside `av2_setup_motion_field()`.
(1) Add checkout step + common setup.
This resolves the issue where the 'Compare hashes' step fails with
'fatal: not a git repository' because the git command was run without
checking out the repository.

(2) Use `github.event.pull_request.head.sha` to get base commit SHA
instead of `github.sha` which is the SHA for the "temporary merge
commit". This ensures that the `git log` command to find commits
containing `STATS_CHANGED` keyword works correctly.

Fixes AOMediaCodec#5106
In the av2_check_profile_interop_conformance() function, the local
variable `profile` is of the BITSTREAM_PROFILE type, so it is redundant
to cast `profile` to BITSTREAM_PROFILE. I suspect the intended cast is
`int`, to match the %d printf format specifier.

Alternatively, we can simply remove the BITSTREAM_PROFILE casts.
…diaCodec#5068) (AOMediaCodec#5069)

This fixes app-only builds where ENABLE_APPS=ON but ENABLE_TESTS,
ENABLE_EXAMPLES, and ENABLE_TOOLS are OFF. avmdec/avmenc still need
libyuv/webm include paths and object sources.

Verified locally:
```
cmake --build build_fast --target avmdec -j8
cmake --build build_fast --target avmenc -j8
```

Fixes AOMediaCodec#5068
Update enable_explicit_ref_frame_map_ setting, and remove duplicate
tests (since we should set enable_explicit_ref_frame_map_ = true for
these tests).

Failures were seen at the lower speed settings (e.g., speed 2), not at
the speed 5 setting used in the test.
As extended sdp is allowed only for partition type less than uneven 4
way search partition, this change has no effect on stats. This is done
for consistency.
This change unifies uneven 4way partition search functions and h
partition search functions into one function

No stats changed.
- Search across ext partitions is rolled into a loop

No stats changed
- Moved pruning using ml results logic to a function
- Removed duplicate/unused initializations
- Handled a case where none partition search gets skipped if
  search_none_after_rect is false and search_none_after_split is true
  and search none is delayed to after rect based on split results
- Moved terminate partition search after none search into none search
  function so that it gets called correctly after none evaluation
- Restored rdmult original in an early return case
The code under CONFIG_COLLECT_PARTITION_DATA is stale. This commit
restores its functionality.
Check the requirement of bitstream conformance that SeenFrameHeader is
not equal to is_first_tile_group. If this requirement is not met, the
decoder may skip the read_uncompressed_header() call in
av2_read_tilegroup_header(), which eventually leads to a null pointer
dereference.

Fixes AOMediaCodec#5142.
The void cast is used to ignore the tp_orig variable in Release builds
because tp_orig is only used in an assert(). So the void cast should be
applied to the variable itself rather than the expression *tp_orig.

This is a port from libaom:
 fd74e18c14 Add the void cast to the tp_orig variable itself

Co-authored-by: Wan-Teh Chang <wtc@google.com>
avmdec and avmenc are no longer added to AVM_EXAMPLE_TARGETS, so we
don't need to filter out avmdec and avmenc in ${AVM_EXAMPLE_TARGETS}.
The read_tile_syntax_info() and read_sequence_tile_info() functions are
defined and only used in av2/decoder/decodeframe.c.
av2_highbd_warp_affine and av2_ext_highbd_warp_affine are called
frequently
during inter prediction at encoder speed 1. Adds NEON intrinsics for
both
functions, achieving 3-5x speedup over C at the kernel level.

Micro-benchmark results (Apple M2 Ultra):

| Kernel | C (ns) | NEON (ns) | Speedup |
|--------|--------|-----------|---------|
| warp 32x32 | 9209 | 1741 | 5.3x |
| warp 16x16 | 2323 | 483 | 4.8x |
| warp 8x8 | 589 | 153 | 3.8x |
| ext_warp 16x16 | 1851 | 441 | 4.2x |
| ext_warp 8x8 | 453 | 158 | 2.9x |

CTC Results (RA, cpu-used=1, 33 frames, classes A4+A5):

| Metric | Delta |
|--------|-------|
| Encode time | -5.5% |
| BD-rate Y | 0.000% |
| BD-rate Cb | 0.000% |
| BD-rate Cr | 0.000% |

Unit tests updated.
Some av2_get_chroma_format_idc() calls in av2/common/level.c and
av2/common/timing.c can be omitted because the callers don't use the
output argument chroma_format_idc.

After these unneeded av2_get_chroma_format_idc() calls are removed,
the subsampling_x, subsampling_y, monochrome parameters of some
functions become unused and need to be removed.
In AV2, monochrome requires subsampling_x == 1 && subsampling_y == 1.
(This somewhat arbitrary constraint comes from AV1.)

Also check the return value of av2_get_chroma_format_idc() at a few call
sites.
This PR enables the 12-bit profile by default and brings its
implementation into conformance with AV2 Spec Draft with 12-bit support:
AOMediaCodec/av2-spec-internal#776.

1. Enable CONFIG_TESTONLY_12BIT_SUPPORT by default
- Flips the 12-bit config default from 0 to 1 so 12-bit encode/decode is
built in by default. (Superseded by the rename below; the flag is now
CONFIG_12BIT_PROFILE.)

2. Align MAIN_4xx_12_IP2 profile with AV2 spec Annex A
- Renames CONFIG_TESTONLY_12BIT_SUPPORT to CONFIG_12BIT_PROFILE, and the
profile enum TEST_ONLY_12BIT_PROFILE to Main_444C_12_IP2 (spec
Main_444C_12_IP2, seq_profile_idc = 5).
- Accepts 8/10/12-bit and all chroma formats (4:0:0/4:2:0/4:2:2/4:4:4)
for profile 5, per Table A.1 (previously 8/10-bit were rejected).
- Sets the max embedded-layer count for profile 5 to 3, matching its
IOP2 limit (Table A.3).
- Gives profile 5 its own PicSize/Bitrate factor row (36 / 3.0) per
Table A.2.
- Updates the in-code Annex A profile/config tables (A.1/A.2/A.5/A.6)
and comments, including the bit_depth_idc mapping.
SamuthirikaS and others added 25 commits August 26, 2026 12:59
Refactor av2_rd_pick_intra_sby_mode() to evaluate Forward Skip
Coding (FSC) modes using an outer fsc_mode loop, eliminating
redundant candidate search logic and unifying non-FSC and
FSC intra mode evaluation.
This patch refactors and abstracts different pruning logics
within the intra mode evaluation loop into a new function and
also avoids redundant computations in intra mode loop.

This is a bit-exact change.
Abstracted code related to multi winner mode processing
into a helper function `refine_winner_intra_mode_tx`.
Also replaces memcpy with `av2_copy_array()`.
This patch renames variables in av2_rd_pick_intra_sby_mode()
for better readablity and code consistency.
- Updated comments to adhere to coding guidelines.
- Used local variables to avoid repetitive access from
data structures.
- Modified code logic for better readability.
Abstracted code related to luma intra mode cost
calculation into a helper function to improve
readability.
The speed feature wienerns_refine_iters was in speed 2.

Enabling it in speed 1 shows speedup with minor quality loss.

Testset          PSNR-YUV   EncTime
A1 (17 frames)    0.02%      95.58%
A2 (33 frames)    0.03%      99.08%

STATS_CHANGED

Change-Id: I91fd3184f45190347c19b49eaf7bf1934238e5ea
… speed 4 (AOMediaCodec#5294)

Skip unit size search for Loop Restoration based on pyramid level for
speed >= 4.

Anchor: e96e164 
Tests: RA, A1 17 frames, others 33 frames

cpu-used=4:
+----------+--------+-------+--------+--------+-------------+----------+
| Summary | Y | U | V | YUV | Enc-time | Dec-time |
+----------+--------+-------+--------+--------+-------------+----------+
| A1 | 0.00% | -0.02% | 0.12% | 0.00% | 95.621% | 99% |
| A2 | 0.00% | -0.11% | 0.02% | 0.00% | 97.579% | 100% |
| A3 | -0.03% | 0.02% | 0.02% | -0.03% | 97.019% | 100% |
| A4 | 0.03% | 0.10% | -0.04% | 0.03% | 97.039% | 100% |
| A5 | -0.04% | 0.20% | -0.09% | -0.03% | 97.165% | 99% |
+----------+--------+--------+--------+--------+------------+----------+
Add a new speed feature to analyze the stationarity of signal to prune
partition type searches.

The speed feature shows a good tradeoff of quality and speed up.

On speed 1, RA:
                Enc_Speedup  PSNR-YUV  Ratio
A1 (17frames)    +4.75%      +0.13%    36.5
A2 (33 frames)   +7.39%      +0.13%    56.8

STATS_CHANGED
- Consolidated initializations and moved early gating
  to the function start
- Introduced local variables for readability
- Renamed calc_masked_type_cost() to calc_comp_type_cost()
  and abstracted compound type cost calculation
- Replaced update_mask_best_mv() with av2_copy_array()
- Renamed rs2 to comp_type_rate
- Simplified the masked_compound_type_rd() signature
- Made cosmetic changes to follow coding guidelines

No Stats Changed
- Added CompTypeRdStats to consolidate RD cost members
  in COMP_RD_STATS
- Replaced local arrays in av2_compound_type_rd() with
  CompTypeRdStats

This improves readability, simplifies the function signature
and data copy in various helper functions.

No Stats Changed
- Unified pick_wedge() and pick_wedge_fixed_sign() to
  avoid code duplication
- Cleaned up the dead code and debug prints in pick_wedge()
- Modified get_wedge_cost() to cleanup unused parameter

No Stats Changed
- Modified comments to ensure correctness and correct punctuations
- Replaced INLINE with AVM_INLINE
- Replaced hard coded values with enums
- Used pre-increments in loop counters

No Stats Changed
Add default extra and encoder configuration options for
AVM_USAGE_REALTIME in av2/av2_cx_iface.c to match standard RTC encoding
defaults, and simplify test/rtc_test.cc to use the new defaults.

Setting `--rt` in the `avmenc` command line applies standard Realtime
(RTC) defaults equivalent to:
  
--bit-depth=8 --passes=1 --lag-in-frames=0 --auto-alt-ref=0
--enable-tpl-model=0 \
--enable-keyframe-filtering=0 --enable-deblocking=1
--enable-restoration=0 \
--enable-intra-edge-filter=0 --enable-flip-idtx=1 --enable-masked-comp=0
\
--enable-onesided-comp=0 --enable-interintra-comp=0
--enable-smooth-interintra=0 \
--enable-diff-wtd-comp=0 --enable-interinter-wedge=0
--enable-interintra-wedge=0 \
--enable-global-motion=0 --enable-warped-motion=0
--enable-smooth-intra=0 \
--enable-paeth-intra=0 --enable-cfl-intra=0 --force-video-mode=1
--enable-overlay=0 \
--enable-angle-delta=0 --enable-trellis-quant=0 --enable-qm=0
--use-intra-dct-only=1 \
--coeff-cost-upd-freq=2 --mode-cost-upd-freq=2 --mv-cost-upd-freq=3
--frame-parallel=0 \
--aq-mode=0 --deltaq-mode=0 --frame-boost=0 --noise-sensitivity=0
--cdf-update-mode=1 \
--reduced-reference-set=1 --enable-ref-frame-mvs=0 --erp-pruning-level=0
\
--use-ml-erp-pruning=0 --enable-ext-partitions=0 --enable-mrls=0
--enable-pc-wiener=0 \
--enable-wiener-nonsep=0 --enable-tip=0 --enable-bawp=0 --enable-cwp=0 \
--enable-imp-msk-bld=0 --enable-ist=0 --enable-inter-ist=0
--enable-inter-ddt=0 \
--enable-cctx=0 --enable-ibp=0 --max-drl-refmvs=0 --max-drl-refbvs=0
--enable-refmvbank=0 \
--enable-opfl-refine=0 --enable-ccso=0 --enable-lf-sub-pu=0
--enable-adaptive-mvd=0 \
--enable-flex-mvres=0 --enable-joint-mvd=0 --enable-refinemv=0
--enable-mvd-sign-derive=0 \
--enable-parity-hiding=0 --enable-warp-delta=0 --enable-warp-extend=0
--tile-columns=0 \
--gf-min-pyr-height=0 --gf-max-pyr-height=0 --enable-sdp=0
--min-gf-interval=8 \
--max-gf-interval=8 --explicit-ref-frame-map=1 --enable-bru=0
--enable-ext-seg=0 \
--enable-six-param-warp-delta=0 --enable-cdef=1
--enable-cdef-on-skip-txfm=0 \
--dpb-size=8 --use-inter-dct-only=0 --enable-fsc=1 --enable-idtx-intra=1
\
--enable-palette=1 --enable-intrabc-ext=0 --enable-intrabc=1
--cpu-used=6 \
    --enable-gdf=0 --max-reference-frames=3
Set tx_size_search_level = USE_FAST_RD, and add level 3 to
restrict_tx_partition_type_search to only test
TX_PARTITION_NONE and TX_PARTITION_SPLIT.

Also switch cdef_pick_method to CDEF_FAST_SEARCH_LVL3, until the
CDEF_PICK_FROM_Q model is further evaluated/improved.

Only affects realtime mode speed >= 6.

This brings back some quality with small speed loss: on average over rtc
set of vga and 720p clips, ~10% bdrate gain with ~4% encoder speed
slowdown. This is with the command line settings below with the qp range
{70 80 90 100 110 120 130 150 170}:

--passes=1 --bit-depth=8 --obu --lag-in-frames=0 --kf-min-dist=1000
--kf-max-dist=1000 --auto-alt-ref=0 --enable-tpl-model=0
--enable-keyframe-filtering=0 --enable-deblocking=1
--enable-restoration=0 --enable-intra-edge-filter=0 --enable-flip-idtx=1
--enable-masked-comp=0 --enable-onesided-comp=0
--enable-interintra-comp=0 --enable-smooth-interintra=0
--enable-diff-wtd-comp=0 --enable-interinter-wedge=0
--enable-interintra-wedge=0 --enable-global-motion=0
--enable-warped-motion=0 --enable-smooth-intra=0 --enable-paeth-intra=0
--enable-cfl-intra=0 --force-video-mode=1 --enable-overlay=0
--enable-angle-delta=0 --enable-trellis-quant=0 --enable-qm=0
--use-intra-dct-only=1 --coeff-cost-upd-freq=2 --mode-cost-upd-freq=2
--mv-cost-upd-freq=3 --frame-parallel=0 --aq-mode=0 --deltaq-mode=0
--frame-boost=0 --noise-sensitivity=0 --cdf-update-mode=1
--reduced-reference-set=1 --enable-ref-frame-mvs=0 --erp-pruning-level=0
--use-ml-erp-pruning=0 --enable-ext-partitions=0 --enable-mrls=0
--enable-pc-wiener=0 --enable-wiener-nonsep=0 --enable-tip=0
--enable-bawp=0 --enable-cwp=0 --enable-imp-msk-bld=0 --enable-ist=0
--enable-inter-ist=0 --enable-inter-ddt=0 --enable-cctx=0 --enable-ibp=0
--max-drl-refmvs=0 --max-drl-refbvs=0 --enable-refmvbank=0
--enable-opfl-refine=0 --enable-ccso=0 --enable-lf-sub-pu=0
--enable-adaptive-mvd=0 --enable-flex-mvres=0 --enable-joint-mvd=0
--enable-refinemv=0 --enable-mvd-sign-derive=0 --enable-parity-hiding=0
--enable-warp-delta=0 --enable-warp-extend=0 --tile-columns=0
--gf-min-pyr-height=0 --gf-max-pyr-height=0 --end-usage=q
--use-fixed-qp-offsets=0 --psnr --threads=1 --enable-sdp=0
--min-gf-interval=8 --max-gf-interval=8 --explicit-ref-frame-map=0
--enable-bru=0 --enable-ext-seg=0 --enable-six-param-warp-delta=0
--enable-cdef=1 --enable-cdef-on-skip-txfm=0 --dpb-size=8
--use-inter-dct-only=0 --enable-fsc=1 --enable-idtx-intra=1
--enable-palette=1 --enable-intrabc-ext=0 --enable-intrabc=0
--tune-content=0 --rt --cpu-used=6 --enable-gdf=0
--max-reference-frames=3 --max-q=qp ---min-q=qp
Promote tx stat pruning to Speed 3
The speed feature was at speed 4.

Enabling the speed feature at speed 3 shows
encoder speed up with acceptable coding loss.

New test on top of base commit
[5d628d8](AOMediaCodec@5d628d8)

Speed 3
Testset                  PSNR-YUV      EncSpeedUp     Ratio
A1 (17 frames)       0.14%              8.28%               59.1
A2 (33 frames)       0.17%              6.53%               38.4

Original test on top of base commit
AOMediaCodec@fe1bfde

Performance on RA speed 3
Testset           PSNR-YUV    EncSpeedUp
A1 (17 frames)    0.15%        6.55%
A2 (33 frames)    0.22%        5.68%

Besides, we tested the speed feature at speed 2, it does not
meet the bar for A1.

Performance on RA speed 2
Testset           PSNR-YUV    EncSpeedUp
A1 (17 frames)    0.16%        4.02%
A2 (33 frames)    0.13%        4.46%

STATS_CHANGED
By reducing the number of candidates, we achieve a considerable
speed up with accepted coding loss.

Result of reducing the count to 3 for speed 4:

TestSet          PSNR-YUV   EncTime    Ratio
A1 (17 frames)    0.10%      96.70%    34.1
A2 (33 frames)    0.07%      96.81%    47.0

Result of reducing the count to 3 for speed 3:

TestSet          PSNR-YUV   EncTime    Ratio
A1 (17 frames)    0.11%      96.87%    29.4
A2 (33 frames)    0.09%      96.41%    41.3

If we reduce the count to 2
The speedup/loss ratio is good for A2, but not good enough for A1.
On speed 4, RA, the performance is:

TestSet          PSNR-YUV   EncTime    Ratio
A1 (17 frames)    0.41%      92.76%    19.0
A2 (33 frames)    0.19%      93.44%    36.9

STATS_CHANGED
…iaCodec#5304)

The wet pass starts mode search from an unbounded rd limit, so its
pruning cannot bite until it finds a good candidate on its own. Start it
instead from the dry pass's rd for the same block and size, plus a
margin.

The two passes see different contexts, so the bound can be violated. A
block left with no valid mode is re-searched once with the original
limit.

Anchor: commit c77f169
Speed 4 (cpu-used=4): FG16 CTC (33 frames, class A1 and A2, RA)

```
  +------------+-------+-------+-------+-------+------+------+
  | Class      |     Y |    Cb |    Cr |  wAvg | Enc% | Dec% |
  +------------+-------+-------+-------+-------+------+------+
  | A1         | -0.30 | -0.29 | -0.38 | -0.30 |  100 |  100 |
  | A2         | -0.17 |  0.07 |  0.31 | -0.15 |  101 |  100 |
  | Avg w/o B2 | -0.21 | -0.04 |  0.11 | -0.19 |  100 |  100 |
  +------------+-------+-------+-------+-------+------+------+
```

STATS_CHANGED
- Introduce the reduce_max_drl_refmvs speed feature to reduce the
default DRL candidate count from 4 to 3 in higher encode speeds.
- Enable frame-level DRL adaptation (allow_frame_max_drl_bits = 1) when
reduce_max_drl_refmvs is active: key, boosted, and low pyramid level
frames (<= 2) retain 4 candidates (max_drl_bits = 3) to retain reference
quality, while leaf frames use 3 candidates (max_drl_bits = 2).
- Condition newmv_drl_search_limit on reduce_max_drl_refmvs to ensure
both ref_mv_idx 0 and 1 are searched before reusing results, preventing
motion predictor diversity collapse.
- The existing --max-drl-refmvs CLI flag can override this behavior.

Evaluation based on 2abcfff with 33 frames at speed 2:
```
+---------+--------+--------+--------+--------+----------+----------+
| Summary |   Y    |   U    |   V    |  YUV   | Enc-time | Dec-time |
+---------+--------+--------+--------+--------+----------+----------+
| A1      | +0.05% | -0.05% | +0.10% | +0.05% |  98.1%   | 100.0%   |
| A2      | +0.08% | +0.27% | -0.03% | +0.08% |  97.5%   | 100.5%   |
+---------+--------+--------+--------+--------+----------+----------+
```
Pulled common calculations from the pixel filtering loop and
conducted them beforehand.

CTC tests showed no stats change.
Added a speed feature to early terminate the warp
delta refinement loop if the rate-distortion cost
improvement in a pass drops below 5%.

Results for RA CTC, A1 17 frames, A2 33 frames,
speed 1: (Anchor: 7c9dfba)

```
+-----+-------+-------+-------+-------+---------------+
|Class|   Y   |   Cb  |   Cr  | wAvg  |EncInstCount(%)|
+-----+-------+-------+-------+-------+---------------+
| A2  |-0.0004|-0.0367|-0.2123|-0.0103| 98.93         |
| A1  |-0.0649| 0.1274| 0.0844|-0.0513| 98.89         |
+-----+-------+-------+-------+-------+---------------+
```
STATS_CHANGED for speed >= 1
Enable frame level bawp based on histogram. Enabled for speed 2 and
above.

33 frame RA results for speed 2 and 3 are as follows: 

+---------+--------+-------+-------+-------+----------+----------+
| Summary | Y | U | V | YUV | Enc-time | Dec-time |
+---------+--------+-------+-------+-------+----------+----------+
Speed2, Overall w/o B2 (RA) +0.00% +0.01% +0.01% +0.00% 99.51% 100.12%
Speed3, Overall w/o B2 (RA) +0.00% +0.01% +0.03% +0.00% 99.45% 99.53%

+---------+--------+-------+-------+-------+----------+----------+
…c#5314)

The expensive pixel-domain distortion path in search_tx_type()
  is skipped if the lightweight transform-domain RD estimate
  already exceeds best_rd or ref_best_rd. This is controlled by
  a new speed feature flag skip_pixel_dist_calc_using_tx_dist,
  enabled for speed >= 1.

  STATS_CHANGED for speed >= 1

 
  Test results (RA) for Speed 1
  Anchor: commit 5d628d8
  A1 - 17 frames
  A2 - 33 frames
    
  ```
  +------------+-------+-------+-------+-------+-------------+
  | Class      |     Y |    Cb |    Cr |  YUV  |EncInstCount%|
  +------------+-------+-------+-------+-------+-------------+
  | A1         | -0.03 |  0.43 | -0.06 |  0.00 |    95.84    |
  | A2         |  0.04 |  0.46 | -0.15 |  0.05 |    97.35    |
  +------------+-------+-------+-------+-------+-------------+
  ```
@yeqing-wu

Copy link
Copy Markdown
Contributor Author

Hi @urvangjoshi, @leolzhao, and @yunqingwang1,

This one is target for av2-enc branch. I've rebased the entire av2-enc branch on top of the main branch.

Meanwhile, I'm running the speed 0/1/4 tests on the rebased branch to confirm nothing is broken, and the results would look as expected.

Feel free to review in parallel and let me know if anything needs changing.

Notes: Please don't squash — we need to preserve the commit history, so rebase only.

Thanks
Yeqing

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.